Freeing vision from frames

3 Volume 3, Issue 1, May 006 The notion of a ‘frame’ of video data has become so embedded in computer vision that it is taken for granted. This is natural, given that the only available input devices have always been frame-based—from drum scanners and videcon tubes to CCDs (chargecoupled devices) and CMOS (complimentary metal-oxide semiconductor) imagers. Also, frame-based imagers have undeniable advantages: they use small pixels, are easy to understand, and are compatible with standard output devices. Are frames the way to go for vision problems, or are they just a holdover from video? Frames carry a heavy penalty: frame-based vision is centered on a stroboscopic series of snapshots taken at a constant rate. The pixels are sampled redundantly, over and over, even if they have nothing novel to say. Bandwidth and dynamic range are limited by the identical sampling rate and integration time. When a human composes a static picture, these may not be terrible disadvantages, but for machine vision in unsupervised environments, the disadvantages of limited dynamic range and sampling rate can be extremely important. Over the past decade, a handful of developers have created novel vision sensor devices that adopt the neuromorphic architecture of redundancy-reduced address-event output. (We don’t have room here to discuss imaging devices that don’t reduce redundancy.) Some of these devices abandon frames altogether. Starting from Mahowald’s address-event representation (AER) silicon retina,1 these new devices offer the promise of more effective ways of tackling real-world vision problems. Mahowald’s AER retina was a demonstration of a concept device that was unusable for any real world task—in fact it was necessary to show it something like a flashing LED to see any sensible response. The University of Pennsylvania’s silicon retina2 marked a major advance by incorporating both sustained and transient types of cells with adaptive spatial and temporal filtering, meaning that the space and time constants vary according to the illumination level and spatio-temporal contrast. This functionality is achieved by the use of tightly coupled log-domain current mode circuits. Of all devices built so far, this one comes closest to capturing key adaptive features of biological retinas. However, the price for this functionality is mismatch: the DC firing rates vary by a factor of 1,000, and one-half of the pixels do not spike at all for moderate contrast. In addition, the use of a passive phototransistor current-gain Freeing vision from frames